{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🚀 Objective-Driven Web Crawling with WaterCrawl, LiteLLM, and Rank-BM25\n",
    "\n",
    "Welcome to this **step-by-step Jupyter Notebook** where we explore goal-oriented web crawling. This tutorial will guide you through using WaterCrawl to search and scrape web content, filter URLs with Rank-BM25 and LLMs, and analyze data to meet a specific objective for a company or website.\n",
    "\n",
    "#### What’s inside?\n",
    "| 🔧 Component | 💡 Why we’re using it |\n",
    "|--------------|----------------------|\n",
    "| **WaterCrawl** | For searching the web and scraping content with precision. |\n",
    "| **LiteLLM** | To interact with various LLM providers for strategy generation, ranking, and content analysis. |\n",
    "| **Rank-BM25** | For efficient keyword-based URL filtering and ranking. |\n",
    "\n",
    "#### Notebook Flow 🗺️\n",
    "1. **Setup**: Install dependencies and configure API keys.\n",
    "2. **Initialization**: Set up the target company or URL and objective.\n",
    "3. **Search Strategy Generation**: Use an LLM to create search strategies.\n",
    "4. **Web Search**: Perform a search with WaterCrawl to gather URLs.\n",
    "5. **URL Filtering**: Apply Rank-BM25 and LLM-based filtering to select relevant URLs.\n",
    "6. **Content Scraping**: Scrape content from top URLs using WaterCrawl.\n",
    "7. **Content Analysis**: Analyze scraped content with LLMs to meet the objective.\n",
    "8. **Results**: Compile and display the final structured response.\n",
    "\n",
    "#### Why you’ll ❤️ this approach\n",
    "- **Efficiency**: Quickly process vast web data.\n",
    "- **Flexibility**: Switch between LLM providers for different tasks.\n",
    "- **Precision**: Combine keyword and semantic analysis for accurate results.\n",
    "- **Customizability**: Tailor search depth, language, and model selection.\n",
    "\n",
    "> **Tip:** If you’re new to WaterCrawl, check out the [WaterCrawl Documentation](https://docs.watercrawl.dev/intro) for more details.\n",
    "\n",
    "Ready? Let’s start by setting up our environment! 🏁"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ➡️ **Install all the dependencies:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ➡️ **API keys you’ll need (grab these first!)** \n",
    "\n",
    "| Service | What it’s for | Where to generate |\n",
    "|---------|---------------|-------------------|\n",
    "| **WaterCrawl** | Auth for search and crawling endpoints | <https://app.watercrawl.dev/dashboard/api-keys> |\n",
    "| **OpenAI/Other LLMs** | LLM interactions | Depends on provider (e.g., OpenAI, Anthropic) |\n",
    "\n",
    "---\n",
    "**Option 1 – Keep it clean: Use a `.env` file** ⚠️\n",
    "\n",
    "Create the file **once**, store your keys, and everything else “just works”.\n",
    "\n",
    "```python\n",
    "# Create .env file\n",
    "env_text = \"\"\"\n",
    "WATERCRAWL_API_KEY=your_watercrawl_api_key_here\n",
    "OPENAI_API_KEY=your_openai_api_key_here\n",
    "ANTHROPIC_API_KEY=your_anthropic_api_key_here\n",
    "DEEPSEEK_API_KEY=your_deepseek_api_key_here\n",
    "\"\"\".strip()\n",
    "\n",
    "with open(\".env\", \"w\") as f:\n",
    "    f.write(env_text)\n",
    "print(\".env file created — now edit it with your real keys ✏️\")\n",
    "```\n",
    "\n",
    "**Option 2 – Quick-and-dirty: Hard-code in the notebook** ⚠️\n",
    "\n",
    "Not recommended — anyone who sees or commits the notebook can read your keys.\n",
    "\n",
    "WATERCRAWL_API_KEY=your_watercrawl_api_key_here\n",
    "LITELLM_API_KEY=your_litellm_api_key_here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ➡️ **If you’re using a `.env` file, load the API keys with dotenv**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "import os\n",
    "\n",
    "load_dotenv()  # pulls everything from .env for any API keys for LLMS\n",
    "\n",
    "WATERCRAWL_API_KEY = os.environ.get(\"WATERCRAWL_API_KEY\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ➡️ **Import necessary packages:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "sys.path.append(os.path.abspath('./objective_crawler'))\n",
    "from objective_crawler.core import ObjectiveCrawler\n",
    "from objective_crawler.clients import WaterCrawler, LLMClient\n",
    "from objective_crawler.config import DEFAULT_SEARCH_LLM, DEFAULT_RANKING_LLM, DEFAULT_CONTENT_ANALYSIS_LLM, DEFAULT_REASONING_LLM, DEFAULT_TOP_K, DEFAULT_NUM_SEARCH_STRATEGIES, DEFAULT_RELEVANCE_THRESHOLD, DEFAULT_SEARCH_DEPTH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ➡️ **Set up the Input and Objective:**\n",
    "Define the company name or website URL you want to search for and the objective you want to achieve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: Tesla\n",
      "Objective: Find information about stock value\n"
     ]
    }
   ],
   "source": [
    "# Example Input and Objective\n",
    "company_or_url = \"Tesla\"\n",
    "objective = \"Find information about stock value\"\n",
    "use_map = False\n",
    "# You can modify these to test with your own inputs\n",
    "print(f\"Input: {company_or_url}\")\n",
    "print(f\"Objective: {objective}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ➡️ **Initialize the Crawler:**\n",
    "Set up the crawler with configurable options for LLM models, number of URLs to scrape, search strategies, and search depth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models\n",
    "\n",
    "<div style=\"display: flex; justify-content: space-between; flex-wrap: wrap; gap: 10px;\">\n",
    "\n",
    "<div style=\"flex: 1; min-width: 200px; background-color: #f0f0f0; padding: 10px; border-radius: 5px;\">\n",
    "\n",
    "**Anthropic Models:**\n",
    "- anthropic/claude-sonnet-4-20250514\n",
    "- anthropic/claude-opus-4-20250514\n",
    "- anthropic/claude-3-7-sonnet-20250219\n",
    "- anthropic/claude-3-5-sonnet-20240620\n",
    "- anthropic/claude-3-sonnet-20240229\n",
    "- anthropic/claude-3-haiku-20240307\n",
    "- anthropic/claude-3-opus-20240229\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"flex: 1; min-width: 200px; background-color: #f0f0f0; padding: 10px; border-radius: 5px;\">\n",
    "\n",
    "**DeepSeek Models:**\n",
    "- deepseek/deepseek-chat\n",
    "- deepseek/deepseek-coder\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"flex: 1; min-width: 200px; background-color: #f0f0f0; padding: 10px; border-radius: 5px;\">\n",
    "\n",
    "**OpenAI Models:**\n",
    "- o4-mini\n",
    "- o3\n",
    "- o3-mini\n",
    "- o1-mini\n",
    "- o1-preview\n",
    "- gpt-4.1\n",
    "- gpt-4.1-mini\n",
    "- gpt-4.1-nano\n",
    "- gpt-4o\n",
    "- gpt-4o-mini\n",
    "\n",
    "</div>\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customizing Your Web Crawling Experience: Key Configuration Options\n",
    "\n",
    "When using an objective-driven web crawling tool, you can fine-tune various settings to optimize how it searches, evaluates, and interprets information. Each setting comes with **default values** defined in a configuration file ([config.py](cci:7://file:///c:/Users/armof/projcects/langchain/watercrawl-tutorials/URL%20and%20objective%20%28Map,%20filter,%20and%20scrape%29/objective_crawler/config.py:0:0-0:0)), which are used automatically if you don’t specify custom values. As a user, you have the flexibility to stick with these defaults for simplicity or override them with new values to tailor the tool to your specific needs. Below, I’ll break down each configuration option in simple terms, highlighting the defaults and the custom settings shown.\n",
    "\n",
    "- **SEARCH_LLM (Default: \"o4-mini\", Custom: \"deepseek/deepseek-chat\")**  \n",
    "  This setting chooses the language model that generates search queries to find relevant information. Imagine it as the creative mind brainstorming the best search terms for your goal. The default is set to \"o4-mini\" in the config, but here it’s customized to DeepSeek's chat model, which excels at crafting diverse and targeted search ideas to kickstart the process.\n",
    "\n",
    "- **RANKING_LLM (Default: \"o4-mini\", Custom: \"anthropic/claude-sonnet-4-20250514\")**  \n",
    "  After gathering potential web pages, this model ranks them based on how likely they are to contain useful content. Think of it as a discerning judge prioritizing the most relevant search results. The default is \"o4-mini,\" but it’s overridden here to Anthropic's Claude Sonnet (version 4-20250514), a powerful choice for accurate and nuanced ranking.\n",
    "\n",
    "- **CONTENT_ANALYSIS_LLM (Default: \"o4-mini\", Custom: \"gpt-4.1-mini\")**  \n",
    "  Once web pages are scraped, this model dives into the content, extracting key details tied to your objective. Picture it as a meticulous researcher summarizing critical insights from each page. By default, it uses \"o4-mini,\" but this example sets it to OpenAI's GPT-4.1 Mini, offering a balance of efficiency and depth for analyzing text.\n",
    "\n",
    "- **REASONING_LLM (Default: \"o4-mini\", Custom: \"o3\")**  \n",
    "  This is the final brain that compiles all the analyzed data into a coherent answer or summary for your objective. It’s like the writer of your final report, weaving together insights from multiple sources. The config defaults to \"o4-mini,\" but it’s changed here to \"o3,\" likely referring to an OpenAI model (possibly a variant or placeholder), chosen for its strong synthesis and reasoning capabilities.\n",
    "\n",
    "- **TOP_K (Default: 5, Custom: \"4\")**  \n",
    "  This number decides how many top-ranked web pages will be deeply analyzed. It’s akin to selecting the top articles to study closely from a broader list. The default in the config is 5, but setting it to 4, as shown, ensures a focused analysis on a slightly smaller, high-quality set of sources, balancing depth with efficiency.\n",
    "\n",
    "- **STRATEGIES (Default: 3, Custom: \"3\")**  \n",
    "  This determines how many different search approaches or query variations the tool will use to find relevant pages. Think of it as asking different experts to search in their unique styles. The default is 3 in the config, and this example keeps it at 3, striking a balance by providing variety in search methods without overwhelming the process.\n",
    "\n",
    "- **SEARCH_DEPTH (Default: \"basic\", Custom: \"basic\")**  \n",
    "  This controls the intensity of the web search—how deep the tool digs into the internet. It’s like choosing between a quick skim or an exhaustive dive. The default is \"basic\" in the config, and it’s left as \"basic\" here, opting for a faster, less intensive search, ideal for straightforward goals, saving time and resources while still delivering results.\n",
    "\n",
    "By tweaking these settings, you can tailor the web crawling tool to match your specific needs—whether you’re aiming for speed, precision, or comprehensive coverage. You can rely on the defaults from the configuration file for a ready-to-go setup or set custom values, as shown in this example, to experiment and optimize the journey from raw web data to actionable insights!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#We can use the default values:\n",
    "# SEARCH_LLM= DEFAULT_SEARCH_LLM\n",
    "# RANKING_LLM= DEFAULT_RANKING_LLM\n",
    "# CONTENT_ANALYSIS_LLM= DEFAULT_CONTENT_ANALYSIS_LLM\n",
    "# REASONING_LLM= DEFAULT_REASONING_LLM\n",
    "# TOP_K=DEFAULT_TOP_K\n",
    "# STRATEGIES=DEFAULT_NUM_SEARCH_STRATEGIES\n",
    "# SEARCH_DEPTH=DEFAULT_SEARCH_DEPTH\n",
    "# ...\n",
    "\n",
    "# Or Selecting the  values \n",
    "SEARCH_LLM= \"deepseek/deepseek-chat\"\n",
    "RANKING_LLM= \"anthropic/claude-sonnet-4-20250514\"\n",
    "CONTENT_ANALYSIS_LLM= \"gpt-4.1-mini\"\n",
    "REASONING_LLM= \"o3\"\n",
    "TOP_K= 5    \n",
    "NUM_SEARCH_STRATEGIES=3\n",
    "SEARCH_DEPTH=\"basic\"\n",
    "MAX_CONTENT_CHARS = 6000  # amount of scraped markdown passed back to GPT\n",
    "DEBUG_MODE=\"False\"\n",
    "# Search-related constants\n",
    "SEARCH_RESULT_LIMIT = 20  # 4 * DEFAULT_TOP_K (get more results initially for filtering)\n",
    "BM25_RESULT_COUNT = 10  # 2 * DEFAULT_TOP_K (intermediate filtering step)\n",
    "SEARCH_DEPTH = \"basic\"  # Options: \"basic\", \"advanced\", \"ultimate\"\n",
    "SEARCH_LANGUAGE = None  # None means auto-detect\n",
    "SEARCH_COUNTRY = None  # None means auto-detect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Initialized WaterCrawler\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Crawler initialized with models:\n",
      "Search LLM: deepseek/deepseek-chat\n",
      "Ranking LLM: anthropic/claude-sonnet-4-20250514\n",
      "Content Analysis LLM: gpt-4.1-mini\n",
      "Reasoning LLM: o3\n",
      "Top K URLs to scrape: 5\n",
      "Number of search strategies: 3\n",
      "Search Depth: basic\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Initialize WaterCrawler with your API key\n",
    "water_crawler = WaterCrawler(api_key=WATERCRAWL_API_KEY)\n",
    "\n",
    "# Create ObjectiveCrawler instance\n",
    "crawler = ObjectiveCrawler(\n",
    "    wc = water_crawler,\n",
    "    top_k =TOP_K,\n",
    "    num_search_strategies = NUM_SEARCH_STRATEGIES,\n",
    "    search_result_limit = SEARCH_RESULT_LIMIT,\n",
    "    bm25_result_count = BM25_RESULT_COUNT,\n",
    "    search_depth = SEARCH_DEPTH,\n",
    "    search_language = SEARCH_LANGUAGE,\n",
    "    search_country = SEARCH_COUNTRY,\n",
    "    debug_mode = DEBUG_MODE,\n",
    "    search_model = SEARCH_LLM,\n",
    "    ranking_model = RANKING_LLM,\n",
    "    reasoning_model = REASONING_LLM,\n",
    "    content_analysis_model = CONTENT_ANALYSIS_LLM  \n",
    ")\n",
    "\n",
    "print(\"Crawler initialized with models:\")\n",
    "print(f\"Search LLM: {SEARCH_LLM}\")\n",
    "print(f\"Ranking LLM: {RANKING_LLM}\")\n",
    "print(f\"Content Analysis LLM: {CONTENT_ANALYSIS_LLM}\")\n",
    "print(f\"Reasoning LLM: {REASONING_LLM}\")\n",
    "print(f\"Top K URLs to scrape: {TOP_K}\")\n",
    "print(f\"Number of search strategies: {NUM_SEARCH_STRATEGIES}\")\n",
    "print(f\"Search Depth: {SEARCH_DEPTH}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ➡️ **Generate Search Strategies with LLM:**\n",
    "Use a dedicated LLM to generate search strategies for finding relevant URLs based on the objective."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92m22:53:33 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= deepseek-chat; provider = deepseek\n",
      "[INFO] \n",
      "LiteLLM completion() model= deepseek-chat; provider = deepseek\n",
      "[INFO] HTTP Request: POST https://api.deepseek.com/beta/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m22:53:41 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Search strategies: ['Tesla stock value', 'site:tesla.com stock price', 'Tesla AND (stock OR share) value']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: SEARCH STRATEGIES GENERATED\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] 1. 'Tesla stock value'\u001b[0m\n",
      "\u001b[35m[DEBUG] 2. 'site:tesla.com stock price'\u001b[0m\n",
      "\u001b[35m[DEBUG] 3. 'Tesla AND (stock OR share) value'\u001b[0m\n",
      "Generated Search Strategies:\n",
      "1. Tesla stock value\n",
      "2. site:tesla.com stock price\n",
      "3. Tesla AND (stock OR share) value\n"
     ]
    }
   ],
   "source": [
    "# Generate search strategies\n",
    "strategies = crawler._derive_search_strategies(objective, company_or_url)\n",
    "\n",
    "print(\"Generated Search Strategies:\")\n",
    "for i, strategy in enumerate(strategies, 1):\n",
    "    print(f\"{i}. {strategy}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ➡️ **Perform Web Search with WaterCrawl and Filter URLs with BM25 :**\n",
    "Use WaterCrawl to search the web based on the generated strategies and filter candidate URLs using BM25."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Search 1/3: 'Tesla stock value'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m[DEBUG] Sending search query 1/3: 'Tesla stock value'\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Search request created with ID: 3af8db0a-25cd-46e6-8cbe-e23b464aa6c0\n",
      "[INFO] Search 'Tesla stock value' returned 20 new results (20 total unique)\n",
      "[INFO] Search 2/3: 'site:tesla.com stock price'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m[DEBUG] Received 20 results, 20 new, total unique URLs: 20\u001b[0m\n",
      "\u001b[35m[DEBUG] Sending search query 2/3: 'site:tesla.com stock price'\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Search request created with ID: 8b76b001-7e61-4ff3-9188-3c98734815eb\n",
      "[INFO] Search 'site:tesla.com stock price' returned 19 new results (39 total unique)\n",
      "[INFO] Search 3/3: 'Tesla AND (stock OR share) value'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m[DEBUG] Received 20 results, 19 new, total unique URLs: 39\u001b[0m\n",
      "\u001b[35m[DEBUG] Sending search query 3/3: 'Tesla AND (stock OR share) value'\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Search request created with ID: b7458f42-37a9-4995-9e33-fc56e98fefa3\n",
      "[INFO] Search 'Tesla AND (stock OR share) value' returned 13 new results (52 total unique)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m[DEBUG] Received 20 results, 13 new, total unique URLs: 52\u001b[0m\n",
      "\u001b[35m================================================================================\n",
      "DEBUG: BM25 URL RANKING\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Found 10 relevant URLs out of 52.\u001b[0m\n",
      "\u001b[35m[DEBUG] Top 10 matches:\u001b[0m\n",
      "\u001b[35m[DEBUG]  1. Score: 5.87 | https://ir.tesla.com/node/16\u001b[0m\n",
      "\u001b[35m[DEBUG]  2. Score: 4.85 | https://tradingeconomics.com/tsla:us\u001b[0m\n",
      "\u001b[35m[DEBUG]  3. Score: 3.75 | https://www.reddit.com/r/OutOfTheLoop/comments/1iyp7dz/whats_up_with_elon_musk_seemingly_not_caring/\u001b[0m\n",
      "\u001b[35m[DEBUG]  4. Score: 3.43 | https://www.ark-invest.com/articles/valuation-models/arks-tesla-model\u001b[0m\n",
      "\u001b[35m[DEBUG]  5. Score: 3.39 | https://www.npr.org/2023/01/06/1146941980/tesla-shares-elon-musk-twitter-electric-cars\u001b[0m\n",
      "\u001b[35m[DEBUG]  6. Score: 3.39 | https://www.ark-invest.com/articles/valuation-models/arks-tesla-price-target-2029\u001b[0m\n",
      "\u001b[35m[DEBUG]  7. Score: 3.14 | https://www.nbcnews.com/business/business-news/tesla-falling-stock-resale-value-elon-musk-trump-what-to-know-rcna196497\u001b[0m\n",
      "\u001b[35m[DEBUG]  8. Score: 3.13 | https://www.calpers.ca.gov/newsroom/calpers-news/2024/calpers-to-vote-against-tesla-ceo-multibillion-dollar-pay-package\u001b[0m\n",
      "\u001b[35m[DEBUG]  9. Score: 2.87 | https://www.cnbc.com/quotes/TSLA\u001b[0m\n",
      "\u001b[35m[DEBUG] 10. Score: 2.86 | https://futurism.com/actual-value-tesla-stock-time-bomb\u001b[0m\n",
      "\u001b[35m================================================================================\n",
      "DEBUG: CANDIDATE URLs AFTER BM25 FILTERING\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Filtered to 10 URLs using BM25\u001b[0m\n",
      "Total URLs from search: 10\n",
      "Sample URLs: ['https://ir.tesla.com/node/16', 'https://tradingeconomics.com/tsla:us', 'https://www.reddit.com/r/OutOfTheLoop/comments/1iyp7dz/whats_up_with_elon_musk_seemingly_not_caring/', 'https://www.ark-invest.com/articles/valuation-models/arks-tesla-model', 'https://www.npr.org/2023/01/06/1146941980/tesla-shares-elon-musk-twitter-electric-cars']\n"
     ]
    }
   ],
   "source": [
    "# Perform web search\n",
    "candidate_urls, all_results = crawler._collect_candidate_urls(company_or_url, use_map, strategies)\n",
    "\n",
    "print(f\"Total URLs from search: {len(candidate_urls)}\")\n",
    "print(\"Sample URLs:\", candidate_urls[:5] if candidate_urls else [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ➡️ **Further Filter URLs with LLM:**\n",
    "Use an LLM to refine the list of URLs based on semantic relevance to the objective."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92m23:06:46 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= claude-sonnet-4-20250514; provider = anthropic\n",
      "[INFO] \n",
      "LiteLLM completion() model= claude-sonnet-4-20250514; provider = anthropic\n",
      "[INFO] HTTP Request: POST https://api.anthropic.com/v1/messages \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:06:53 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Top 5 candidate pages:\n",
      "[INFO] https://ir.tesla.com/node/16                                  1.00  Official Tesla investor relations page - primary source for authoritative stock and financial information directly from the company\n",
      "[INFO] https://www.cnbc.com/quotes/TSLA                              0.95  CNBC quotes page specifically for TSLA stock ticker - provides real-time stock price, charts, and financial data\n",
      "[INFO] https://tradingeconomics.com/tsla:us                          0.90  Financial data platform with dedicated TSLA stock page showing current value, historical data, and market analysis\n",
      "[INFO] https://www.ark-invest.com/articles/valuation-models/arks-te  0.85  Professional investment firm's Tesla valuation model with specific price targets - highly relevant for stock value analysis\n",
      "[INFO] https://www.ark-invest.com/articles/valuation-models/arks-te  0.80  ARK Invest's comprehensive Tesla valuation model providing detailed analysis of stock value fundamentals\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: URL RANKING RESULTS\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] 1. https://ir.tesla.com/node/16\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 1.00 | Reason: Official Tesla investor relations page - primary source for authoritative stock and financial information directly from the company\u001b[0m\n",
      "\u001b[35m[DEBUG] 2. https://www.cnbc.com/quotes/TSLA\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 0.95 | Reason: CNBC quotes page specifically for TSLA stock ticker - provides real-time stock price, charts, and financial data\u001b[0m\n",
      "\u001b[35m[DEBUG] 3. https://tradingeconomics.com/tsla:us\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 0.90 | Reason: Financial data platform with dedicated TSLA stock page showing current value, historical data, and market analysis\u001b[0m\n",
      "\u001b[35m[DEBUG] 4. https://www.ark-invest.com/articles/valuation-models/arks-tesla-price-target-2029\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 0.85 | Reason: Professional investment firm's Tesla valuation model with specific price targets - highly relevant for stock value analysis\u001b[0m\n",
      "\u001b[35m[DEBUG] 5. https://www.ark-invest.com/articles/valuation-models/arks-tesla-model\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 0.80 | Reason: ARK Invest's comprehensive Tesla valuation model providing detailed analysis of stock value fundamentals\u001b[0m\n",
      "URLs after LLM filtering: 5\n",
      "Top URLs for scraping: ['https://ir.tesla.com/node/16', 'https://www.cnbc.com/quotes/TSLA', 'https://tradingeconomics.com/tsla:us', 'https://www.ark-invest.com/articles/valuation-models/arks-tesla-price-target-2029', 'https://www.ark-invest.com/articles/valuation-models/arks-tesla-model']\n"
     ]
    }
   ],
   "source": [
    "# Filter URLs with LLM\n",
    "final_url_list = crawler._rank_links(candidate_urls, objective, company_or_url)\n",
    "\n",
    "print(f\"URLs after LLM filtering: {len(final_url_list)}\")\n",
    "print(\"Top URLs for scraping:\", final_url_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ➡️ **Scrape Content of Top URLs and Analyze its content, finaly get final answer leveraging Reasoning_llm  :**\n",
    "Scrape the content of the selected URLs using WaterCraw, then call _analyze_page_content to have the content_llm analyze the content, then get the final answer leveraging reasoning_llm calling _get_final_answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92m23:16:54 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= o3; provider = openai\n",
      "[INFO] \n",
      "LiteLLM completion() model= o3; provider = openai\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: FINAL RESULT GENERATION\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Successful analyses: 5\u001b[0m\n",
      "\u001b[35m[DEBUG] Failed analyses: 0\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:17:19 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n"
     ]
    }
   ],
   "source": [
    "analysis_results = []\n",
    "search_results_dict = {r.get(\"url\", \"\"): r for r in all_results if \"url\" in r} if not use_map else {}\n",
    "\n",
    "for i, page_url in enumerate(final_url_list, 1):\n",
    "    try:\n",
    "        markdown_content = crawler.scrape(page_url)\n",
    "        if markdown_content is None:\n",
    "            raise ValueError(f\"Scrape returned None for URL: {page_url}\")\n",
    "        analysis_result = crawler._analyze_page_content(markdown_content, objective, page_url)\n",
    "    except Exception as e:\n",
    "        description = search_results_dict.get(page_url, {}).get(\"description\", \"No description available\")\n",
    "        analysis_result = {\n",
    "            \"verified_url\": page_url,\n",
    "            \"objective\": objective,\n",
    "            \"result_of_analysis\": f\"URL could not be crawled: {page_url}. Description: {description}\"\n",
    "        }\n",
    "    analysis_results.append(analysis_result)\n",
    "\n",
    "# Store individual analyses in metadata for debugging or transparency\n",
    "result = crawler._generate_final_result(analysis_results, objective)\n",
    "result[\"_metadata\"] = {\"individual_analyses\": analysis_results}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### **Printing the Results**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"objective_fulfilled\": true,\n",
      "  \"final_answer\": \"Available public sources provide both real-time market data for Tesla\\u2019s stock (ticker: TSLA) and independent valuation targets:\\n\\n1. Current / Real-Time Market Value\\n   \\u2022 CNBC\\u2019s quote page and TradingEconomics both stream the live price of TSLA together with historical charts, fundamental ratios and news feeds.  These pages are intended for investors who need the up-to-the-second trading value.\\n\\n2. Independent Valuation Models\\n   \\u2022 ARK Invest 2029 Target \\u2013 ARK estimates a base-case price of roughly $2,600 per share for Tesla in 2029, with a bear case near $2,000 and a bull case near $3,100.\\n   \\u2022 ARK Invest 2026 Target \\u2013 In a separate model, ARK\\u2019s expected value per share for 2026 is approximately $4,600, based on its open-source assumptions about vehicle sales, autonomous ride-hail revenue, and margins.\\n\\n3. Company Investor-Relations Context\\n   \\u2022 Tesla\\u2019s own Investor-Relations FAQ notes that the company does not comment on daily price moves or give investment advice and explains how shareholders can obtain stock certificates.\\n\\nIn short, investors can check the live TSLA price on outlets such as CNBC or TradingEconomics, while forward-looking valuations such as ARK Invest\\u2019s models suggest potential future share prices in the $2,000\\u2013$4,600 range depending on the time horizon and scenario.\",\n",
      "  \"reference_urls\": [\n",
      "    \"https://www.cnbc.com/quotes/TSLA\",\n",
      "    \"https://tradingeconomics.com/tsla:us\",\n",
      "    \"https://www.ark-invest.com/articles/valuation-models/arks-tesla-price-target-2029\",\n",
      "    \"https://www.ark-invest.com/articles/valuation-models/arks-tesla-model\",\n",
      "    \"https://ir.tesla.com/node/16\"\n",
      "  ],\n",
      "  \"pages_analyzed\": 5,\n",
      "  \"successful_pages\": 5,\n",
      "  \"_metadata\": {\n",
      "    \"individual_analyses\": [\n",
      "      {\n",
      "        \"verified_url\": \"https://ir.tesla.com/node/16\",\n",
      "        \"objective\": \"Find information about stock value\",\n",
      "        \"result_of_analysis\": \"URL could not be crawled: https://ir.tesla.com/node/16. Description: Can Tesla comment on a move in its share price or provide investment advice? ... Can I obtain a stock certificate for my Tesla shares? Who is Tesla's\\u00a0...\"\n",
      "      },\n",
      "      {\n",
      "        \"verified_url\": \"https://www.cnbc.com/quotes/TSLA\",\n",
      "        \"objective\": \"Find information about stock value\",\n",
      "        \"result_of_analysis\": \"URL could not be crawled: https://www.cnbc.com/quotes/TSLA. Description: Get Tesla Inc (TSLA:NASDAQ) real-time stock quotes, news, price and financial information from CNBC.\"\n",
      "      },\n",
      "      {\n",
      "        \"verified_url\": \"https://tradingeconomics.com/tsla:us\",\n",
      "        \"objective\": \"Find information about stock value\",\n",
      "        \"result_of_analysis\": \"URL could not be crawled: https://tradingeconomics.com/tsla:us. Description: Tesla stock price, live market quote, shares value, historical data, intraday chart, earnings per share and news.\"\n",
      "      },\n",
      "      {\n",
      "        \"verified_url\": \"https://www.ark-invest.com/articles/valuation-models/arks-tesla-price-target-2029\",\n",
      "        \"objective\": \"Find information about stock value\",\n",
      "        \"result_of_analysis\": \"URL could not be crawled: https://www.ark-invest.com/articles/valuation-models/arks-tesla-price-target-2029. Description: ARK\\u2019s price target for Tesla is $2,600 per share in 2029 with our bear and bull cases valued between ~$2,000 and $3,100 per share.\"\n",
      "      },\n",
      "      {\n",
      "        \"verified_url\": \"https://www.ark-invest.com/articles/valuation-models/arks-tesla-model\",\n",
      "        \"objective\": \"Find information about stock value\",\n",
      "        \"result_of_analysis\": \"URL could not be crawled: https://www.ark-invest.com/articles/valuation-models/arks-tesla-model. Description: ARK\\u2019s updated open-source Tesla model yields an expected value per share of $4,600 in 2026. Please explore and test our assumptions.\"\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "pretty_json = json.dumps(result, indent=2)\n",
    "print(pretty_json)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[96mTHE REASONING LLM ANSWER (FINAL ANSWER OF THE WHOLE ANALYSIS):\u001b[0m\n",
      "\n",
      "\u001b[92mAvailable public sources provide both real-time market data for Tesla’s stock (ticker: TSLA) and independent valuation targets:\n",
      "\n",
      "1. Current / Real-Time Market Value\n",
      "   • CNBC’s quote page and TradingEconomics both stream the live price of TSLA together with historical charts, fundamental ratios and news feeds.  These pages are intended for investors who need the up-to-the-second trading value.\n",
      "\n",
      "2. Independent Valuation Models\n",
      "   • ARK Invest 2029 Target – ARK estimates a base-case price of roughly $2,600 per share for Tesla in 2029, with a bear case near $2,000 and a bull case near $3,100.\n",
      "   • ARK Invest 2026 Target – In a separate model, ARK’s expected value per share for 2026 is approximately $4,600, based on its open-source assumptions about vehicle sales, autonomous ride-hail revenue, and margins.\n",
      "\n",
      "3. Company Investor-Relations Context\n",
      "   • Tesla’s own Investor-Relations FAQ notes that the company does not comment on daily price moves or give investment advice and explains how shareholders can obtain stock certificates.\n",
      "\n",
      "In short, investors can check the live TSLA price on outlets such as CNBC or TradingEconomics, while forward-looking valuations such as ARK Invest’s models suggest potential future share prices in the $2,000–$4,600 range depending on the time horizon and scenario.\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "class Colors:\n",
    "    OKCYAN = '\\033[96m'\n",
    "    OKGREEN = '\\033[92m'\n",
    "    ENDC = '\\033[0m' # Resets the color\n",
    "    BOLD = '\\033[1m'\n",
    "    \n",
    "print(Colors.BOLD + Colors.OKCYAN + 'THE REASONING LLM ANSWER (FINAL ANSWER OF THE WHOLE ANALYSIS):' + Colors.ENDC)\n",
    "print()\n",
    "print(Colors.OKGREEN + result['final_answer'] + Colors.ENDC)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ➡️ **We could have done all above in One Step!**\n",
    "In the above cells, we walked you trhough the step by step process of how the crawler works. We could have done all above in one step, just by calling the run method of the crawler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Objective: Find information about stock value for Tesla\n",
      "\u001b[92m23:47:43 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= deepseek-chat; provider = deepseek\n",
      "[INFO] \n",
      "LiteLLM completion() model= deepseek-chat; provider = deepseek\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: OBJECTIVE CRAWLER STARTED\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Objective: Find information about stock value\u001b[0m\n",
      "\u001b[35m[DEBUG] Target: Tesla\u001b[0m\n",
      "\u001b[35m[DEBUG] Search Depth: basic\u001b[0m\n",
      "\u001b[35m[DEBUG] Search Language: None\u001b[0m\n",
      "\u001b[35m[DEBUG] Search Country: None\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.deepseek.com/beta/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:47:49 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Search strategies: ['Tesla stock value', 'site:tesla.com stock price', 'Tesla AND (stock OR share) AND (value OR price)']\n",
      "[INFO] Search 1/3: 'Tesla stock value'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: SEARCH STRATEGIES GENERATED\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] 1. 'Tesla stock value'\u001b[0m\n",
      "\u001b[35m[DEBUG] 2. 'site:tesla.com stock price'\u001b[0m\n",
      "\u001b[35m[DEBUG] 3. 'Tesla AND (stock OR share) AND (value OR price)'\u001b[0m\n",
      "\u001b[35m[DEBUG] Sending search query 1/3: 'Tesla stock value'\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Search request created with ID: 9c9ad5bc-906b-4bc4-957b-d22b0b8b6917\n",
      "[INFO] Search 'Tesla stock value' returned 20 new results (20 total unique)\n",
      "[INFO] Search 2/3: 'site:tesla.com stock price'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m[DEBUG] Received 20 results, 20 new, total unique URLs: 20\u001b[0m\n",
      "\u001b[35m[DEBUG] Sending search query 2/3: 'site:tesla.com stock price'\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Search request created with ID: 47045202-5923-456b-ab54-571df7acc226\n",
      "[INFO] Search 'site:tesla.com stock price' returned 19 new results (39 total unique)\n",
      "[INFO] Search 3/3: 'Tesla AND (stock OR share) AND (value OR price)'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m[DEBUG] Received 20 results, 19 new, total unique URLs: 39\u001b[0m\n",
      "\u001b[35m[DEBUG] Sending search query 3/3: 'Tesla AND (stock OR share) AND (value OR price)'\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Search request created with ID: efce42d6-01dd-447f-843c-c6c6fd51b729\n",
      "[INFO] Search 'Tesla AND (stock OR share) AND (value OR price)' returned 10 new results (49 total unique)\n",
      "\u001b[92m23:48:06 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= claude-sonnet-4-20250514; provider = anthropic\n",
      "[INFO] \n",
      "LiteLLM completion() model= claude-sonnet-4-20250514; provider = anthropic\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m[DEBUG] Received 20 results, 10 new, total unique URLs: 49\u001b[0m\n",
      "\u001b[35m================================================================================\n",
      "DEBUG: BM25 URL RANKING\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Found 10 relevant URLs out of 49.\u001b[0m\n",
      "\u001b[35m[DEBUG] Top 10 matches:\u001b[0m\n",
      "\u001b[35m[DEBUG]  1. Score: 6.17 | https://tradingeconomics.com/tsla:us\u001b[0m\n",
      "\u001b[35m[DEBUG]  2. Score: 6.13 | https://ir.tesla.com/node/16\u001b[0m\n",
      "\u001b[35m[DEBUG]  3. Score: 4.54 | https://www.ark-invest.com/articles/valuation-models/arks-tesla-price-target-2027\u001b[0m\n",
      "\u001b[35m[DEBUG]  4. Score: 4.14 | https://www.reddit.com/r/OutOfTheLoop/comments/1iyp7dz/whats_up_with_elon_musk_seemingly_not_caring/\u001b[0m\n",
      "\u001b[35m[DEBUG]  5. Score: 3.73 | https://www.npr.org/2023/01/06/1146941980/tesla-shares-elon-musk-twitter-electric-cars\u001b[0m\n",
      "\u001b[35m[DEBUG]  6. Score: 3.59 | https://www.cnbc.com/quotes/TSLA\u001b[0m\n",
      "\u001b[35m[DEBUG]  7. Score: 3.49 | https://finance.yahoo.com/quote/TSLA/\u001b[0m\n",
      "\u001b[35m[DEBUG]  8. Score: 3.49 | https://www.cnn.com/markets/stocks/TSLA\u001b[0m\n",
      "\u001b[35m[DEBUG]  9. Score: 3.49 | https://www.calpers.ca.gov/newsroom/calpers-news/2024/calpers-to-vote-against-tesla-ceo-multibillion-dollar-pay-package\u001b[0m\n",
      "\u001b[35m[DEBUG] 10. Score: 3.45 | https://www.tradingview.com/symbols/NASDAQ-TSLA/\u001b[0m\n",
      "\u001b[35m================================================================================\n",
      "DEBUG: CANDIDATE URLs AFTER BM25 FILTERING\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Filtered to 10 URLs using BM25\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.anthropic.com/v1/messages \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:48:15 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Top 5 candidate pages:\n",
      "[INFO] https://finance.yahoo.com/quote/TSLA/                         1.00  Yahoo Finance is a premier financial data platform with dedicated stock quote pages. The '/quote/TSLA/' path directly indicates Tesla stock information with real-time pricing, charts, and comprehensive financial data.\n",
      "[INFO] https://www.cnbc.com/quotes/TSLA                              0.95  CNBC is a leading financial news network with dedicated stock quote sections. The '/quotes/TSLA' path specifically targets Tesla stock information with current pricing and market data.\n",
      "[INFO] https://tradingeconomics.com/tsla:us                          0.90  Trading Economics specializes in financial and economic data. The 'tsla:us' identifier directly references Tesla's US stock ticker with comprehensive market analysis and historical data.\n",
      "[INFO] https://www.tradingview.com/symbols/NASDAQ-TSLA/              0.85  TradingView is a professional trading platform. The '/symbols/NASDAQ-TSLA/' path specifically identifies Tesla stock on NASDAQ with advanced charting and technical analysis tools.\n",
      "[INFO] https://ir.tesla.com/node/16                                  0.80  Official Tesla investor relations subdomain provides authoritative financial information. While the '/node/16' path is not descriptive, IR pages typically contain stock performance data and official company financial metrics.\n",
      "[INFO] [1/5] Crawling and analyzing: https://finance.yahoo.com/quote/TSLA/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: URL RANKING RESULTS\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] 1. https://finance.yahoo.com/quote/TSLA/\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 1.00 | Reason: Yahoo Finance is a premier financial data platform with dedicated stock quote pages. The '/quote/TSLA/' path directly indicates Tesla stock information with real-time pricing, charts, and comprehensive financial data.\u001b[0m\n",
      "\u001b[35m[DEBUG] 2. https://www.cnbc.com/quotes/TSLA\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 0.95 | Reason: CNBC is a leading financial news network with dedicated stock quote sections. The '/quotes/TSLA' path specifically targets Tesla stock information with current pricing and market data.\u001b[0m\n",
      "\u001b[35m[DEBUG] 3. https://tradingeconomics.com/tsla:us\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 0.90 | Reason: Trading Economics specializes in financial and economic data. The 'tsla:us' identifier directly references Tesla's US stock ticker with comprehensive market analysis and historical data.\u001b[0m\n",
      "\u001b[35m[DEBUG] 4. https://www.tradingview.com/symbols/NASDAQ-TSLA/\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 0.85 | Reason: TradingView is a professional trading platform. The '/symbols/NASDAQ-TSLA/' path specifically identifies Tesla stock on NASDAQ with advanced charting and technical analysis tools.\u001b[0m\n",
      "\u001b[35m[DEBUG] 5. https://ir.tesla.com/node/16\u001b[0m\n",
      "\u001b[35m[DEBUG]    Score: 0.80 | Reason: Official Tesla investor relations subdomain provides authoritative financial information. While the '/node/16' path is not descriptive, IR pages typically contain stock performance data and official company financial metrics.\u001b[0m\n",
      "\u001b[35m[DEBUG] Crawling URL 1/5: https://finance.yahoo.com/quote/TSLA/\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92m23:48:28 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n",
      "[INFO] \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: CONTENT ANALYSIS FOR: https://finance.yahoo.com/quote/TSLA/\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Original content length: 6175 chars\u001b[0m\n",
      "\u001b[35m[DEBUG] Content sent to LLM: 6000 chars (truncated: True)\u001b[0m\n",
      "\u001b[35m===== CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n",
      "\u001b[35m[Content truncated to 2000 chars]\u001b[0m\n",
      "\u001b[35mOops, something went wrong\n",
      "\n",
      "NasdaqGS - BOATS Real Time Price • USD\n",
      "\n",
      "# Tesla, Inc. (TSLA)\n",
      "\n",
      "Follow\n",
      "\n",
      "Add holdings \n",
      "\n",
      "[ TSLA: Risk or rebound? ](https://stockstory.org/us/stocks/nasdaq/tsla?partner=yahoo&utm_source=yahoo&utm_medium=financequotetopnavigation&utm_campaign=quoteunderperformdesktop)\n",
      "\n",
      "293.94 \n",
      "\n",
      "-21.41 \n",
      "\n",
      "(-6.79%) \n",
      "\n",
      "At close: July 7 at 4:00:01 PM EDT \n",
      "\n",
      "297.02 \n",
      "\n",
      "+3.08 \n",
      "\n",
      "+(1.05%) \n",
      "\n",
      "Overnight: 2:48:06 AM EDT \n",
      "\n",
      "### \n",
      "\n",
      "This price reflects trading activity during the overnight session on the Blue Ocean ATS, available 8 PM to 4 AM ET, Sunday through Thursday, when regular markets are closed. \n",
      "\n",
      "[ TSLA: Risk or rebound? ](https://stockstory.org/us/stocks/nasdaq/tsla?partner=yahoo&utm_source=yahoo&utm_medium=financequotetopnavigation&utm_campaign=quoteunderperformdesktop)\n",
      "\n",
      "1D \n",
      "\n",
      "5D \n",
      "\n",
      "### -9.17%\n",
      "\n",
      "1M \n",
      "\n",
      "### -0.41%\n",
      "\n",
      "6M \n",
      "\n",
      "### -25.46%\n",
      "\n",
      "YTD \n",
      "\n",
      "### -27.21%\n",
      "\n",
      "1Y \n",
      "\n",
      "### 16.87%\n",
      "\n",
      "5Y \n",
      "\n",
      "### 217.23%\n",
      "\n",
      "All \n",
      "\n",
      "### 23,105.78%\n",
      "\n",
      "Key Events \n",
      "\n",
      "Mountain\n",
      "\n",
      "[ Advanced Chart ](/chart/TSLA)\n",
      "\n",
      "Loading Chart for TSLA\n",
      "\n",
      "Chart does not reflect overnight price.\n",
      "\n",
      "  * Previous Close 315.35 \n",
      "  * Open 291.37 \n",
      "  * Bid 290.00 x 100\n",
      "  * Ask 305.96 x 100\n",
      "  * Day's Range 288.77 - 296.15 \n",
      "  * 52 Week Range 182.00 - 488.54 \n",
      "  * Volume 130,122,655 \n",
      "  * Avg. Volume 116,733,008 \n",
      "  * Market Cap (intraday) 946.769B \n",
      "  * Beta (5Y Monthly) 2.46\n",
      "  * PE Ratio (TTM) 167.01 \n",
      "  * EPS (TTM) 1.76 \n",
      "  * Earnings Date Jul 23, 2025\n",
      "  * Forward Dividend & Yield \\--\n",
      "  * Ex-Dividend Date \\--\n",
      "  * 1y Target Est 306.07 \n",
      "\n",
      "\n",
      "\n",
      "##  Tesla, Inc. Overview Auto Manufacturers / Consumer Cyclical\n",
      "\n",
      "Tesla, Inc. designs, develops, manufactures, leases, and sells electric vehicles, and energy generation and storage systems in the United States, China, and internationally. The company operates in two segments, Automotive; and Energy Generation and Storage. The Automotive segment offers electric vehicles, as well as sells automotive regulatory credits; and non-warranty after-sales vehicle, used vehicles, body shop and parts, supercharging, retail merchandise, and ...\u001b[0m\n",
      "\u001b[35m===== END CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:48:35 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] [2/5] Crawling and analyzing: https://www.cnbc.com/quotes/TSLA\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m===== LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35mRelevant: Tesla, Inc. (TSLA) stock closed at $293.94 on July 7, 2025, down $21.41 (-6.79%). Overnight price was $297.02 (+1.05%). Key stats include a market cap of $946.77B, 52-week range of $182.00 - $488.54, volume of 130.1M shares, and average volume of 116.7M. Financial metrics: trailing P/E 167.01, EPS 1.76, forward P/E 151.52, PEG ratio 5.14, price/sales 10.77, price/book 12.68. Profit margin is 6.38%, return on assets 3.72%, return on equity 8.77%, revenue $95.72B, net income $6.11B, total cash $37B, debt/equity 17.41%, and levered free cash flow $3.36B. Earnings date is July 23, 2025, with a 1-year target estimate of $306.07.\u001b[0m\n",
      "\u001b[35m===== END LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35m[DEBUG] Crawling URL 2/5: https://www.cnbc.com/quotes/TSLA\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92m23:48:46 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n",
      "[INFO] \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: CONTENT ANALYSIS FOR: https://www.cnbc.com/quotes/TSLA\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Original content length: 9059 chars\u001b[0m\n",
      "\u001b[35m[DEBUG] Content sent to LLM: 6000 chars (truncated: True)\u001b[0m\n",
      "\u001b[35m===== CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n",
      "\u001b[35m[Content truncated to 2000 chars]\u001b[0m\n",
      "\u001b[35mSkip Navigation\n",
      "\n",
      "[![logo](https://static-redesign.cnbcfm.com/dist/2469ed0a9a4cafdf055e.svg)](/)\n",
      "\n",
      "[Markets](/markets/)\n",
      "\n",
      "[Business](/business/)\n",
      "\n",
      "[Investing](/investing/)\n",
      "\n",
      "[Tech](/technology/)\n",
      "\n",
      "[Politics](/politics/)\n",
      "\n",
      "[Video](/tv/)\n",
      "\n",
      "[Watchlist](/watchlist/)\n",
      "\n",
      "[Investing Club](/investingclub/subscribe?__source=investingclub|globalnav|join&tpcc=investingclub|globalnav|join)\n",
      "\n",
      "![Join IC](https://static-redesign.cnbcfm.com/dist/93743f20be95b721880f.svg)\n",
      "\n",
      "[PRO](/application/pro?__source=pro|globalnav|join&tpcc=pro|globalnav|join)\n",
      "\n",
      "![Join Pro](https://static-redesign.cnbcfm.com/dist/69ae09b80acd376e9c97.svg)\n",
      "\n",
      "[Livestream](/live-tv/)\n",
      "\n",
      "Menu\n",
      "\n",
      "#  Tesla Inc TSLA:NASDAQ\n",
      "\n",
      "EXPORT ![download chart](https://static-redesign.cnbcfm.com/dist/e20a86c8296962374986.svg)\n",
      "\n",
      "WATCHLIST+\n",
      "\n",
      "RT Quote | Last NYSE Arca, VOL From CTA | USD\n",
      "\n",
      "Last | 07/03/25 EDT\n",
      "\n",
      "293.94![quote price arrow down](https://static-redesign.cnbcfm.com/dist/4ee243ff052e81044388.svg)-21.41 (-6.79%)\n",
      "\n",
      "Volume\n",
      "\n",
      "118,378,280\n",
      "\n",
      "52 week range\n",
      "\n",
      "182.00 \\- 488.54\n",
      "\n",
      "![Loading...](https://static-redesign.cnbcfm.com/dist/7bfb09d729f4a90936a2.svg)\n",
      "\n",
      "  * Open291.37\n",
      "  * Day High296.15\n",
      "  * Day Low288.77\n",
      "  * Prev Close293.94\n",
      "  * 52 Week High488.54\n",
      "  * 52 Week High Date12/18/24\n",
      "  * 52 Week Low182.00\n",
      "  * 52 Week Low Date08/05/24\n",
      "\n",
      "\n",
      "\n",
      "ALL CNBCINVESTING CLUBPRO\n",
      "\n",
      "  * [Tariff letters from Trump, like those of lovers, seem to signify little](https://www.cnbc.com/2025/07/08/cnbc-daily-open-trump-tariff-letters-seem-to-signify-little.html \"Tariff letters from Trump, like those of lovers, seem to signify little\")19 Min AgoCNBC.com\n",
      "\n",
      "  * [Trump’s tariff letters set the heart racing, but don’t seem to promise anything new](https://www.cnbc.com/2025/07/08/cnbc-daily-open-tariff-letters-set-the-heart-racing-but-dont-seem-new.html \"Trump's tariff letters set the heart racing, but don't seem to promise anything new\")5 Hours AgoCNBC.com\n",
      "\n",
      "  * [CoreWeave, Core Scientific shares fall after firms confirm $9 billion deal: CNBC Crypto World![CNBC Video](https://static-redesign....\u001b[0m\n",
      "\u001b[35m===== END CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:48:52 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] [3/5] Crawling and analyzing: https://tradingeconomics.com/tsla:us\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m===== LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35mRelevant: Tesla Inc (TSLA) stock last traded at $293.94 on 07/03/25, down $21.41 (-6.79%) with a volume of 118,378,280 shares. The 52-week range is $182.00 to $488.54, with the high on 12/18/24 and low on 08/05/24. Key stats include Market Cap of $946.768B, Shares Outstanding 3.22B, 10-day average volume 110.28M, Beta 2.34, and YTD % change -27.21. Financial ratios: EPS (TTM) 1.82, P/E (TTM) 161.78, Forward P/E (NTM) 147.26, EBITDA (TTM) $12.651B, ROE (TTM) 9.16%, Revenue (TTM) $95.724B, Gross Margin 17.66%, Net Margin 6.72%, Debt to Equity 10.09%. Upcoming earnings date is 07/23/2025.\u001b[0m\n",
      "\u001b[35m===== END LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35m[DEBUG] Crawling URL 3/5: https://tradingeconomics.com/tsla:us\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92m23:49:00 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n",
      "[INFO] \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: CONTENT ANALYSIS FOR: https://tradingeconomics.com/tsla:us\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Original content length: 20599 chars\u001b[0m\n",
      "\u001b[35m[DEBUG] Content sent to LLM: 6000 chars (truncated: True)\u001b[0m\n",
      "\u001b[35m===== CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n",
      "\u001b[35m[Content truncated to 2000 chars]\u001b[0m\n",
      "\u001b[35m#  Tesla | TSLAStock Price | Live Quote | Historical Chart\n",
      "\n",
      "  * Chart \n",
      "  * Quotes \n",
      "  * Financials \n",
      "  * Alerts \n",
      "  * [__Export]()\n",
      "    * ![CSV download button](https://d3fy651gv2fhd3.cloudfront.net/images/downloadicons/download-csv-filled.svg) Download Data\n",
      "    * ![Excel download button](https://d3fy651gv2fhd3.cloudfront.net/images/downloadicons/microsoft-excel-filled.svg) Excel Add-in\n",
      "    * ![API download button](https://d3fy651gv2fhd3.cloudfront.net/images/downloadicons/download-api.svg) API Access\n",
      "\n",
      "\n",
      "\n",
      "Stock Price \n",
      "\n",
      "293.94\n",
      "\n",
      "Daily Change \n",
      "\n",
      "-21.41  -6.79% \n",
      "\n",
      "Monthly \n",
      "\n",
      "-4.74% \n",
      "\n",
      "Yearly \n",
      "\n",
      "16.09% \n",
      "\n",
      "Q3 Forecast \n",
      "\n",
      "305.29\n",
      "\n",
      "Search...\n",
      "\n",
      "1D \n",
      "\n",
      "Compare\n",
      "\n",
      "__\n",
      "\n",
      "Export\n",
      "\n",
      "__API\n",
      "\n",
      "Created with Highcharts 10.1.0Context Menu 150200250300350400450500550\n",
      "\n",
      "  *   *   *     *   *     *     *     *   *     *   *     *   *     *   *   *   *     * \n",
      "\n",
      "\n",
      "SepNov2025MarMayJul\n",
      "\n",
      "293.94\n",
      "\n",
      "1D1W1M6M1Y5Y10YAll\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Tesla (USD) 293.94 -21.41 (-6.79%)\n",
      "\n",
      "Tesla X \n",
      "\n",
      "LNR\n",
      "\n",
      "null:cur X \n",
      "\n",
      "LNR\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Intraday1 Minute5 Minutes15 Minutes1 HourDays1 Day1 Week1 Month\n",
      "\n",
      "Submit\n",
      "\n",
      "  \n",
      "\n",
      "\n",
      "Peers | Price |  | Chg | Day | Year | Date  \n",
      "---|---|---|---|---|---|---  \n",
      "[ Adient ](/adnt:us) | 22.00  |  | -0.38  | -1.70% | -8.18% | Jul/07   \n",
      "[ Albemarle ](/alb:us) | 65.50  |  | -1.71  | -2.54% | -33.98% | Jul/07   \n",
      "[ Autoliv ](/alv:us) | 116.32  |  | -1.42  | -1.21% | 7.72% | Jul/07   \n",
      "[ Aptiv ](/aptv:us) | 72.56  |  | 0.37  | 0.51% | 3.22% | Jul/03   \n",
      "[ BMW ](/bmw:gr) | 78.06  |  | 0.22  | 0.28% | -10.40% | Jul/07   \n",
      "[ Borgwarner ](/bwa:us) | 34.84  |  | -0.48  | -1.36% | 10.83% | Jul/07   \n",
      "[ Avis Budget ](/car:us) | 181.52  |  | 0.21  | 0.12% | 77.93% | Jul/07   \n",
      "[ Continental ](/con:gr) | 75.00  |  | -0.02  | -0.03% | 24.25% | Jul/07   \n",
      "[ Wolfspeed ](/cree:us) | 2.31  |  | 1.13  | 95.76% | -90.55% | Jul/07   \n",
      "[ Mercedes-Benz AG ](/dai:gr) | 49.90  |  | -0.18  | -0.36% | -23.69% | Jul/07   \n",
      "[ Dana ](/dan:us) | 17.49  |  | -0.33  | -1.85% | 53.42% | Jul/07   \n",
      "[ Enphase Energy ](/enph:us) | 42.46  |  | -0.11  | -0.26% | -58.70% | Jul/07   \n",
      "[ Ford Moto...\u001b[0m\n",
      "\u001b[35m===== END CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:49:05 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] [4/5] Crawling and analyzing: https://www.tradingview.com/symbols/NASDAQ-TSLA/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m===== LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35mRelevant: Tesla stock price is $293.94 as of Monday, July 7th, down $21.41 (-6.79%) from the previous session. Monthly change is -4.74%, yearly change is +16.09%. Q3 forecast price is $305.29, with a one-year forecast of $277.00. The data includes intraday and historical price charts, daily and yearly percentage changes, and peer stock prices for comparison.\u001b[0m\n",
      "\u001b[35m===== END LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35m[DEBUG] Crawling URL 4/5: https://www.tradingview.com/symbols/NASDAQ-TSLA/\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92m23:49:18 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n",
      "[INFO] \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: CONTENT ANALYSIS FOR: https://www.tradingview.com/symbols/NASDAQ-TSLA/\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Original content length: 39132 chars\u001b[0m\n",
      "\u001b[35m[DEBUG] Content sent to LLM: 6000 chars (truncated: True)\u001b[0m\n",
      "\u001b[35m===== CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n",
      "\u001b[35m[Content truncated to 2000 chars]\u001b[0m\n",
      "\u001b[35mMain content\n",
      "\n",
      "[ ](/)\n",
      "\n",
      "Search\n",
      "\n",
      "EN  __\n",
      "\n",
      "[Get started](/pricing/?source=header_go_pro_button&feature=start_free_trial)\n",
      "\n",
      "![](https://s3-symbol-logo.tradingview.com/tesla.svg)\n",
      "\n",
      "TSLAMarket closed\n",
      "\n",
      "293.94RUSD\n",
      "\n",
      "−21.41−6.79%\n",
      "\n",
      "[See on Supercharts](/chart/?symbol=NASDAQ%3ATSLA)\n",
      "\n",
      "[](/chart/?symbol=NASDAQ%3ATSLA)\n",
      "\n",
      "![Tesla](https://s3-symbol-logo.tradingview.com/tesla--big.svg)![Tesla](https://s3-symbol-logo.tradingview.com/tesla--big.svg)![Tesla](https://s3-symbol-logo.tradingview.com/tesla--big.svg)\n",
      "\n",
      "# Tesla\n",
      "\n",
      "TSLA![](https://s3-symbol-logo.tradingview.com/source/NASDAQ.svg)Nasdaq Stock Market\n",
      "\n",
      "TSLA![](https://s3-symbol-logo.tradingview.com/source/NASDAQ.svg)Nasdaq Stock Market\n",
      "\n",
      "TSLA![](https://s3-symbol-logo.tradingview.com/source/NASDAQ.svg)Nasdaq Stock Market\n",
      "\n",
      "TSLA![](https://s3-symbol-logo.tradingview.com/source/NASDAQ.svg)Nasdaq Stock Market\n",
      "\n",
      "Market closedMarket closed\n",
      "\n",
      "293.94USDR\n",
      "\n",
      "−21.41−6.79%\n",
      "\n",
      "At close at 00:00 GMT\n",
      "\n",
      "USD\n",
      "\n",
      "No trades\n",
      "\n",
      "[See on Supercharts](/chart/?symbol=NASDAQ%3ATSLA)\n",
      "\n",
      "Overview \n",
      "\n",
      "[ Financials ](/symbols/NASDAQ-TSLA/financials-overview/ \"Tesla, Inc. Financial statements\") [ News ](/symbols/NASDAQ-TSLA/news/ \"Tesla, Inc. News\") [ Ideas ](/symbols/NASDAQ-TSLA/ideas/) [ Discussions ](/symbols/NASDAQ-TSLA/minds/ \"Minds on TSLA\") [ Technicals ](/symbols/NASDAQ-TSLA/technicals/ \"Tesla, Inc. Technical analysis\") [ Forecast ](/symbols/NASDAQ-TSLA/forecast/ \"TSLA price target\") [ Seasonals ](/symbols/NASDAQ-TSLA/seasonals/) [ Options ](/symbols/NASDAQ-TSLA/options/ \"TSLA options\") [ Bonds ](/symbols/NASDAQ-TSLA/bonds/)\n",
      "\n",
      "## [TSLA chart﻿ ﻿](/chart/?symbol=NASDAQ%3ATSLA)\n",
      "\n",
      "[](/widget-docs/widgets/charts/symbol-overview/ \"Get widget\")\n",
      "\n",
      "[Full chart](/chart/?symbol=NASDAQ%3ATSLA)\n",
      "\n",
      "[](/chart/?symbol=NASDAQ%3ATSLA \"Full chart\")\n",
      "\n",
      "|  [](https://www.tradingview.com/chart/?symbol=NASDAQ%3ATSLA) |   \n",
      "---|---|---  \n",
      "|  |   \n",
      "  \n",
      "1 day−6.79%5 days−8.12%1 month−1.64%6 months−25.20%Year to date−24.65%1 year17.67%5 years213.82%All time‪23.11 K‬%\n",
      "\n",
      "## Key facts today\n",
      "\n",
      "﻿\n",
      "\n",
      "William Blair analys...\u001b[0m\n",
      "\u001b[35m===== END CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:49:22 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] [5/5] Crawling and analyzing: https://ir.tesla.com/node/16\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m===== LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35mRelevant: Tesla (TSLA) stock closed at $293.94 USD, down $21.41 (−6.79%). Market capitalization is $945.46 billion. Price-to-earnings (TTM) ratio is 173.42, with basic EPS (TTM) of $1.99. Net income for the fiscal year is $7.13 billion, and revenue is $97.69 billion. Shares float is 2.80 billion, and beta (1Y) is 1.75. Upcoming earnings report for Q2 2025 has an EPS estimate of $0.42 and revenue estimate of $22.74 billion. Recent analyst downgrade from Outperform to Market Perform due to removal of EV tax credits and sales challenges; Tesla sold 721,000 vehicles in H1 2025, a 13% drop year-over-year, with projected total sales of 1.7 million for 2025.\u001b[0m\n",
      "\u001b[35m===== END LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35m[DEBUG] Crawling URL 5/5: https://ir.tesla.com/node/16\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[92m23:49:30 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n",
      "[INFO] \n",
      "LiteLLM completion() model= gpt-4.1-mini; provider = openai\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m================================================================================\n",
      "DEBUG: CONTENT ANALYSIS FOR: https://ir.tesla.com/node/16\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Original content length: 11718 chars\u001b[0m\n",
      "\u001b[35m[DEBUG] Content sent to LLM: 6000 chars (truncated: True)\u001b[0m\n",
      "\u001b[35m===== CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n",
      "\u001b[35m[Content truncated to 2000 chars]\u001b[0m\n",
      "\u001b[35m[Skip to main content](main-content)\n",
      "\n",
      "Stock Show All Hide All\n",
      "\n",
      "  * How do I purchase shares of Tesla? Do you have a Direct Stock Purchase Plan?\n",
      "\n",
      "Tesla’s shares trade on the NASDAQ exchange, under the ticker symbol TSLA. To purchase shares, you will need to do so through a broker. If you do not have a brokerage account, you will need to open one. At this time, Tesla does not have a direct stock purchase program.\n",
      "\n",
      "  * Can Tesla comment on a move in its share price or provide investment advice?\n",
      "\n",
      "Unfortunately, we cannot comment on moves in our share price or offer investment advice.\n",
      "\n",
      "  * Does Tesla pay a dividend? Does it plan to?\n",
      "\n",
      "Tesla has never declared dividends on our common stock. We intend on retaining all future earnings to finance future growth and therefore, do not anticipate paying any cash dividends in the foreseeable future. \n",
      "\n",
      "  * When was Tesla's initial public offering (IPO)?\n",
      "\n",
      "Tesla's initial public offering was on June 29, 2010. The IPO was priced at $17 per share.\n",
      "\n",
      "  * Can I obtain a stock certificate for my Tesla shares? Who is Tesla's transfer agent?\n",
      "\n",
      "The transfer agent and registrar for Tesla’s common stock is Computershare Trust Company located at 250 Royall Street in Canton, MA 02021. Their telephone number is (800) 662-7232. Please contact Computershare for any questions about receiving stock certificates for your shares.\n",
      "\n",
      "  * For SolarCity shareholders, what is the stock basis impact of the merger for tax purposes?\n",
      "\n",
      "Please see IRS Form 8937: [Form-8937](https://digitalassets.tesla.com/tesla-contents/image/upload/IR/Form_8937)\n",
      "\n",
      "  * For Maxwell shareholders, what is the stock basis impact of the merger for tax purposes?\n",
      "\n",
      "Please see IRS Form 8937: [Form-8937](https://digitalassets.tesla.com/tesla-contents/image/upload/IR/Form-8937---Maxwell-Transaction)\n",
      "\n",
      "  * What are the CUSIP Numbers for Tesla's stock and convertible bonds?\n",
      "\n",
      "The CUSIP number for Tesla’s common stock is 88160R101.  \n",
      "  \n",
      "For Tesla’s convertible notes outstanding, the CUSIP numbers ar...\u001b[0m\n",
      "\u001b[35m===== END CONTENT SENT TO LLM FOR ANALYSIS =====\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:49:35 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "\u001b[92m23:49:35 - LiteLLM:INFO\u001b[0m: utils.py:3225 - \n",
      "LiteLLM completion() model= o3; provider = openai\n",
      "[INFO] \n",
      "LiteLLM completion() model= o3; provider = openai\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35m===== LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35mRelevant: Tesla’s shares trade on NASDAQ under ticker TSLA; no direct stock purchase plan exists, shares must be bought via a broker. IPO was on June 29, 2010, priced at $17 per share. Tesla has never paid dividends and does not anticipate paying any in the foreseeable future. The CUSIP number for common stock is 88160R101. Quarterly financial report dates and SEC filings, which include stock value information, are posted at ir.tesla.com and sec.gov. Tesla does not comment on share price movements or provide investment advice.\u001b[0m\n",
      "\u001b[35m===== END LLM ANALYSIS RESULT =====\u001b[0m\n",
      "\u001b[35m================================================================================\n",
      "DEBUG: FINAL RESULT GENERATION\n",
      "================================================================================\u001b[0m\n",
      "\u001b[35m[DEBUG] Successful analyses: 5\u001b[0m\n",
      "\u001b[35m[DEBUG] Failed analyses: 0\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "\u001b[92m23:49:57 - LiteLLM:INFO\u001b[0m: utils.py:1236 - Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Wrapper: Completed Call, calling success_handler\n",
      "[INFO] Execution time: 134.39 seconds\n"
     ]
    }
   ],
   "source": [
    "result_of_run = crawler.run(company_or_url, objective)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[36mTHE REASONING LLM ANSWER (FINAL ANSWER OF THE WHOLE ANALYSIS):\u001b[0m\n",
      "\n",
      "\u001b[92mCurrent Tesla, Inc. (TSLA) stock snapshot (as of market close on Monday, July 7, 2025):\n",
      "\n",
      "Price & Daily Movement\n",
      "• Closing price: $293.94\n",
      "• Daily change: −$21.41 (−6.79%)\n",
      "• After-hours/overnight: $297.02 (+1.05%)\n",
      "\n",
      "Trading Range & Volume\n",
      "• 52-week range: $182.00 – $488.54\n",
      "• Day’s volume: ~118–130 million shares (vs. 10-day avg. ~110 million; 3-month avg. ~116 million)\n",
      "\n",
      "Market Capitalization & Shares\n",
      "• Market cap: ≈$946 billion\n",
      "• Shares outstanding/float: ~3.22 billion / 2.80 billion\n",
      "\n",
      "Key Valuation Multiples\n",
      "• Trailing P/E: 161–173\n",
      "• Forward P/E (NTM): 147–152\n",
      "• PEG ratio: 5.14\n",
      "• Price/Sales: 10.77\n",
      "• Price/Book: 12.68\n",
      "\n",
      "Profitability & Financial Metrics\n",
      "• EPS (TTM): $1.76–1.99\n",
      "• Net margin: 6.4–6.7 %\n",
      "• Return on equity: ~9 %\n",
      "• Revenue (TTM): $95.7–97.7 billion\n",
      "• Net income (TTM): $6.1–7.1 billion\n",
      "• EBITDA (TTM): $12.65 billion\n",
      "• Gross margin: 17.7 %\n",
      "• Cash on hand: ≈$37 billion\n",
      "• Debt/Equity: 10–17 %\n",
      "• Levered free cash flow: $3.36 billion\n",
      "\n",
      "Beta & Performance\n",
      "• 1-year beta: 1.75–2.34 (high volatility)\n",
      "• Monthly price change: −4.74 %\n",
      "• 1-year price change: +16.09 %\n",
      "• Year-to-date change (2025): −27.21 %\n",
      "\n",
      "Forecasts & Analyst Notes\n",
      "• Consensus 1-year target price: ~$277–306 per share\n",
      "• Q3 2025 model price (TradingEconomics): $305.29\n",
      "• Next earnings release: July 23, 2025 (Q2 2025; est. EPS $0.42 on revenue $22.74 billion)\n",
      "• Recent analyst action: Downgraded to “Market Perform” citing lost EV tax credits and 13 % YoY drop in H1 2025 vehicle deliveries (721 k units; full-year projection 1.7 million).\n",
      "\n",
      "Corporate Details\n",
      "• Ticker: TSLA (NASDAQ)\n",
      "• IPO date & price: June 29, 2010 at $17 per share\n",
      "• Dividend policy: No dividends paid or planned\n",
      "• CUSIP: 88160R101\n",
      "• Shares must be purchased through a broker; no direct purchase plan.\n",
      "\n",
      "Summary\n",
      "Tesla’s share price sits just under $294 with a market cap near $950 billion, trading at high valuation multiples (P/E >160) despite recent price weakness (−27 % YTD). Profitability remains positive with ~6–7 % net margin and healthy cash reserves, but sales headwinds and regulatory changes have led to cautious analyst sentiment and a modest downside consensus target (~$277–306). Key upcoming catalyst is the Q2 2025 earnings report on July 23, 2025.\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "class Colors:\n",
    "    OKCYAN = '\\033[96m'\n",
    "    DARKCYAN = '\\033[36m'\n",
    "    GREEN = '\\033[92m'\n",
    "    OKGREEN = '\\033[92m'\n",
    "    ENDC = '\\033[0m' # Resets the color\n",
    "    BOLD = '\\033[1m'\n",
    "    \n",
    "print(Colors.BOLD + Colors.DARKCYAN + 'THE REASONING LLM ANSWER (FINAL ANSWER OF THE WHOLE ANALYSIS):' + Colors.ENDC)\n",
    "print()\n",
    "print(Colors.GREEN + result_of_run['final_answer'] + Colors.ENDC)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🌟 Conclusion\n",
    "Congratulations! You've successfully used WaterCrawl, LiteLLM, and Rank-BM25 to search the web, filter URLs, scrape content, and analyze it to meet a specific objective for a company or website. \n",
    "\n",
    "#### What you’ve learned:\n",
    "- How to set up and configure tools for web data extraction.\n",
    "- Generating search strategies and performing web searches with WaterCrawl.\n",
    "- Filtering URLs with BM25 and LLMs for relevance.\n",
    "- Scraping and analyzing content to answer targeted questions.\n",
    "\n",
    "#### Next Steps:\n",
    "- Experiment with different companies, URLs, and objectives.\n",
    "- Try different LLM models for each task (search, ranking, analysis, reasoning).\n",
    "- Scale up by adjusting search depth or integrating with other tools.\n",
    "\n",
    "If you found this tutorial helpful, consider starring the [WaterCrawl repo](https://github.com/watercrawl/watercrawl) on GitHub! ⭐"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
